Observations using the Fermi Large Area Telescope (Fermi-LAT) have found a significant gamma-ray excess surrounding the center of the Milky Way (GC). One possible interpretation of this excess invokes gamma-ray emission from an undiscovered population of either young or recycled pulsars densely clustered throughout the inner kiloparsec of the Milky Way. While these systems, by construction, have individual fluxes that lie below the point source sensitivity of the Fermi-LAT, they may already be observed in multiwavelength observations. Notably the Australia Telescope National Facility (ATNF) catalog of radio pulsars includes 270 sources observed in the inner 10 degrees around the GC. We calculate the gamma-ray emission observed from these 270 sources and obtain three key results: (1) point source searches in the GC region produce a plethora of highly significant gamma-ray "hotspots", compared to searches far from the Galactic plane, (2) there is no statistical correlation between the positions of these gamma-ray hotspots and the locations of ATNF pulsars, and (3) the spectrum of the most statistically significant gamma-ray hotspots is substantially softer than the spectrum of the GC gamma-ray excess. These results place strong constraints on models where young pulsars produce the majority of the gamma-ray excess, and disfavor some models where millisecond pulsars produce the gamma-ray excess.